CARP with SR

So I've just come up against this myself and was going to post my findings, figure I'll post them here instead of starting a new thread - it sounds like my issue is related.

I've just clean installed PFSense 2.5 on two of my hypervisors, intending to set them up for High Availability to replace an old cisco router that I don't want to spend the money to replace.

Both hypervisors are running Hyper-V 2019 with Intel X520 10gb nics - on one of them I'm running the Windows Server NIC teaming in an Active/Standby configuration - 10gb link active, 1gb standby - as you may be aware this configuration does not allow SR IOV to work.

On the other I'm using the newer 'Switch Embedded Teaming' mode which allows SR IOV, with 10gb/10gb active/active (switch capacity is the reason I'm only running this on one hypervisor at the moment, SET mode prevents an Active/Standby setup)

On both PFSense virtual machines I set the (virtual) NICS to enable MAC Spoofing, which as I understood from the documentation should be all that was required for such a configuration to work on Hyper-V.

In my testing, having set up the CARP virtual IP addresses, what I found was that on the SET teamed hypervisor, I couldn't contact it on the CARP ip address - it could ping itself on this address but nothing else could, which suggested some kind of connectivity issue rather than a setup issue (especially as it worked fine on the other hypervisor with the synced configuration when I failed over). The other hypervisor worked as expected.

Looking at the SET team documentation, there is mention here that the MAC address is overwritten when the virtual switch is in dynamic load balancing mode: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v-virtual-switch/rdma-and-switch-embedded-teaming#bkmk_mac

I'm not certain that I created the switch in Dynamic mode although it seems fairly likely - so I switched the mode across to Hyper V Port distribution, which specifically states there "No source MAC replacement is done" - but still no joy (I've not rebooted the hypervisor, it's possible that it's required to change the mode across but I would have thought unlikely).

Next thing I read that 'Receive Segment Coalescing' has caused some people problems in Hyper-V 2019, so I turned that off for the VSwitch, no joy there either (https://docs.microsoft.com/en-us/windows-server/networking/technologies/hpn/rsc-in-the-vswitch) - as an aside, this is still enabled on the other hypervisor and no issues with it there.

Now, one point to make is that enabling MAC spoofing prevents your virtual NICS from using SR IOV or VMQ acceleration, which is rather far from ideal in any case, as with heavy traffic usage this will at best increase CPU usage, and at worst limit the throughput.

So I'm not sure what is the best way forward now. I could just remove the switch embedded teaming from this hypervisor and drop back to the windows NIC teaming that I've been using previously, although in principle switch embedded teaming is a better option as it permits SR IOV and so should improve my network performance for the other virtual servers on this hypervisor - my broad plan had been to increase switch capacity and bring all my hypervisors across to SET.